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Abstract 

A meta-analysis of outcomes from studies investigating pretest 
effects was conducted. All outcomes were computed as standardized 
differences between pretested and non-pretested groups. Eleven other 
variables were coded for each outcome. Initial descriptive statis- 
tics were indicative of differences between randomized and nonran- 
domized studies, so all further analyses were based on randomized 
group outcomes (n s 110). For all outcomes the average effect size 
was +.34, indicating the general elevating effect of pretest on 
posttest. Cognitive outcomes were raised .51, attitude outcomes 
.33, and others about .27 standard deviations. Seventy percent 
of all effects were positive, and 93% of the cognitive effects were 
positive. Duration of time between pre- and post-testing was not 
related to effect size, although effect size is generally smaller 
for durations over one month. Year of publication, sample size 
presence of experimental treatment, and sameness or difference of 
pretest and posttest were not significantly related to effect size. 
Researchers must continue to include pretest as a design variable 
when it is present, and to estimate its effect. 
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A Meta-analysis of Pretest 
Sensitization Effects in 
Experimental Design 

Incroduction 

The pretest is a common condition cf experimental and nonexperi- 
mental research. Campbell and Stanly (1963) discussed it in their 
landmark review of experimental design considerations. Bracht and 
Glass (1968) and Wfclch and Walberg (1970) have presented more recent 
reviews of the effect of pretesting on posttest scores. Jaeger (1975) 
has advocated the use of protests for many evaluation and research 
applications. The independent experimental contribution of pretests 
to dependent variables is unknown. Welch and Walberg's (1970) sum- 
mary listed previous results in the usual review format of signifi- 
cance or non-significance. They concluded the long-term cognitive 
effects are small or nil, while there may be short term effects. 
These effects, they suggested, are greater for attitude tests than 
for cognitive tests. 

The recent work of Glass (1976) and his colleagues and students 
in statistical review of research, termed meta-analysis, allows a 
more systematic analysis of the pretest experimental effect. Instead 
of relying on binary decisions about significance, the pretest effect 
is examined as a standardized difference between pretested and non- 
pretested groups. The purpose of the study is to estimate the pretest 
effect utilizing meta-analytic techniques and to investigate modera- 

§ 

tor variables associated with the potential pretest effect. 
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Procedures 

Literature Search 

In order to explore pretest sensitization and to determine 
the nature of its effect the Meta -Ana lysis techniques for inte- 
grating existing research findings were utilized as proposed by 
Glass (1977). Briefly surtmarized, this approach requires that 
either all studies or a sufficiently large representative sample 
on a given topic be located. Each study is treated as an indi- 
vidualized unit of analysis with multiple dependent variables 
and/or multiple comparison groups considered as separate units. 

The method utilized in compiling the literature for ex- 
amination was two-fold. The Automated Information-Retrieval 
Service (AIRS) at Texas A&M was employed to retrieve relevant 
information from the Educational Resources Information Center 
(ERIC), Psychological Abstracts, and the Social Science Cita- 
tion Index (SOCIAL SCISEARCH). Secondly, references cited in 
the Welch and Walberg (1970) article were reviewed and a branch- 
ing technique was utilized in an effort to locate all possible 
literature of interest to the present study. 

When the task of locating the material for study was com- 
plete, the research reports were screened for appropriateness 
of content, and articles focusing on outcomes other than human 
educational and psychological phenomena were omitted (e.g. 
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animal studies, social interaction studies, neonatal response 
studies). Also, studies in which all groups were pretested 
were omitted. 

Variables Coded 

The variables extracted from each study in the literature 
search were year of publication, subjects' grade in school or 
age, sex of subjects (all males, all females, or mixed), duration 
of time betweeen pretest and post-test, citation (or not) of 
test reliability, sample size, randomization in selection of 
subjects (or not), random assignment of subjects to groups (or 
not), presence or absence of relevant treatment between pretest 
and posttest, category of dependent variable, similarity or dis- 
similarity of pretest and posttest, and effect size of pretest on 
posttest. The variables were chosen on the basis of theoretical 
relevance to the pretest effect, interest for experimental design, 
and utility in previous meta-analysis. For example, year of 
publication has been shown by Smith and Glass (1977) and Glass, 
Smith, and Barton (Note 1) to be related to effect size in 
several areas of research. Smith and Glass (1977) showed a simi- 
lar effect for duration of treatment on psychoterapy outcomes. 
They also compared randomized with non-randomized experiments, 
demonstrating no difference in psychotherapy outcome effect sizes. 

Dependent variables in each study were first coded by name 
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of test or'scale, but it soon became apparent that a larger 
categorization was needed. Most outcomes were easily cate- 
gorized as cognitive (achievement test score) or attitudinal . 
While four other categories were utimately distinguishable 
(physical performance, rating of performance, projective score, 
and choice preference). The number of outcomes for these was 
found to be small compared with those of the first two categories. 

Duration of time between pre-and post testing was categor- 
ized ex post facto into four categories: hours, with duration of 
two hours or less; days , with duration less than seven; weeks , 
with duration less than two, and months , with duration less than 
or equal to nine. No studies researched in this paper investi- 
gated pretest effects for duration greater than nine months. 

Effect sizes were computed from summary statistics reported 
by authors. Group means were required in all cases except when 
t-statistics (or F-statistics with one degree of freedom) were 
reported. Group sample sizps were also necessary. Each ex- 
perimental design was reduced to one or more of the following 
forms (see Campbell and Stanley, 1963) • 
I. R 0 XO 1 1 . R 0 0 



R 



XO 



0 



III. 



0 XO 



IV. 



0 0 



xo 



0 
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Thus if no comparison of group means conformed to type 
I -IV, the study was ignored. F^r each form I -IV the effect 
size was defined as 

effect size = X protested - X pretested , (1) 

^not pretested group 

When the standard deviation of the not-pretested group was un- 
available and not recoverable, a pooled variance statistic was 
used. If mean square error in a study was available, its square 
root served as standard deviation. If a t-statistic or F-statis- 
tic was reported the following transformation allowed estimation 
of the effect size: 

Effect = t (l/n,H l/n 2 ) ? * , (2) 
,F = t, 

wheren^ng = sample sizes for the two groups. 

It is possible some bias results in such cases if the the pre- 
test affects variance of posttest scores, A test of this hypo- 
thesis was made by forming an F-ratio of pretested group variance 
divided by unpretested group variance for twenty-three identifi- 
able cases. Two of the twenty- three (8.7%) were significant at 
p = .05, indicating no variance bias. In some designs it was 
possible to estimate mean square error from the means of two or 
more control groups, none of which was pretested (all were re- 
quired to be of the same form for estimation). Mean square be- 
tween these groups was computed and the square root taken to be 
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an estimate of group variability in the absense of pretest effect. 
The effect was then computed using equation (2). Irrelevant 
crossing variables were eliminated in factorial designs by 
averaging across them to obtain marginal means appropriate to 
forms I to IV. 

In some designs several effects were computable utilizing 
different groups. This was commonly the result of several dif- 
ferent treatments being administered. Each possible effect was 
coded in these designs. When several dependent variables were 
examined in a study a separate effect was computed for each de- 
pendent variable. This resulted in a few studies yielding a 
large number of effects. The number of effects per study varied 
from one to twenty- three. In some studies dependent variables 
were all of one category (eg. attitude) which in others dependent 
variable represented two or three categories of outcome. 

The sjgn^ of the effect size is of great importance to inter- 
pretation of results. For cognitive outcomes a positive effect 
implies a higher test score for the pretested group. For atti- 
tude outcomes positive effect was defined as a more favorable 
attitude toward the subject. Sign of the effects was reversed 
to fit this definition. For behaviors the positive direction 
reflected the experimental aim of the study; for example avoid- 
ance behaviors should decrease -'n a systematic desensitization 
for fear of snakes. Thus, fewer avoidance behaviors was a 
positive outcome in this case. In a curriculum study the 
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the number of activities of high school students related to 
science was counted. An increase in this number was taken as a 
positive outcome. 

Data Analysis 

The first analysis was the extraction of descriptive statis- 
tics. Of special importance are the distributions of effects for 
randomized and non-randomized studies. If the distributions are 
statistically nondistinguishable then they may be pooled for sub- 
sequent analyses, as Smith and Glass (1977) did for psychotherapy 
outcomes. If different, the nonrandomized effects are presumed 
to have an unknown bias and are not further investigated. 

In either case further analysis is used to examine the re- 
lationship of each coded variable to effect size, singly and 
multiply. Correlation, multiple regression and analysis of 
variance are all applied, depending on the nominal or interval 
character of the variables, 

Results 

Twenty-nine separate studies were analyzed which all had as 
a primary focus examination of the pretest effect. From these 
studies 140 separate effects were extracted and coded with the 
other independent variables, Summary statistics for these 
effects are given in Table 1, along with summary statistics for 
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subsets of the studies based on randomized or nonrandomized 
assignments and to pretest conditions. The nonrandomized 
effects have a different mer,n (t = 2.44, df = 81.8, p<.02, 
Satterthwaite's approximation using unequal variances) and 
different variance (F = 3.03; df * 1.09.29, p< .01) from the non- 
randomized effects. Consequently, it is concluded that nonrandom- 
ized study effects are biased systematically and are excluded 
from further analysis. 

Insert Table 1 about here 

Summary statistics for randomized study effects are given 
in Table 2. Effects of subjects who received a treatment (Form I) 
had a mean of .31, not significantly different from untreated 
subject effects, mean .39. Both distributions are positively 
skewed. 

A subset of the pretest sensitization 1 iterature has been con- 
cerned with the interaction of pretest condition and treatment. 
Globally, this hypothesized interaction does not appear to be 
large if it exists at ail, since the effect size for treated and 
untreated groups are so similar. 

The time between pretest and posttest as measured in hours, 
days, weeks and months is not linearly related to effect size. 
The pattern of means suggests small effects for delays of less 
than a day or over one months, but there are nany gaps in the 



ERIC 



11 



A Meta-analysis of Pretest 

10 

intervals observed in the studies. Most short delays observed were 
of duration less than one hour. All studies with delays over a few 
hours and less than week utilized two or three day durations. Other 
studies used delays of seven, ten, and eleven and twelve days, while 
longer term studies used two week, thirty-five days, forty-five days, 
ten weeks, ninety-one days, and nine months. A systematic investi- 
gation of delay time between pretest and posttest would help to 
clarify the effect size. 

Category of outcome yields some slightly surprising results. 
Cognitive effects under pretest conditions are over one half standard 
deviation above nonpretested groups, and 93% of the effects are posi- 
tive. For attitude outcomes the effect mean is about .33 with 62% of 
the effects positive. This is a reversal of the conclusion of Bracht 
and Glass (1968) and Welch and Walberg (1970). Closer examination 
of the cognitive outcomes provides some useful results, given in 
Table 3. No clear conclusion about delay between pretest and post- 
test is possible, given the large effects which occurred for duration 
of one week's order. Pretest effect appears to be larger for pretest 
similar to posttest, and pretest effect is much larger for untreated 
groups than for treated groups. 

For attitude outcomes many conclusions made about cognitive 
outcomes are reversed. Pretest effect appears larger for delays on 
the order of two days. Pretest effect is greater on different con- 
tent posttests, and there is little difference in effect size between 
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treated and untreated groups. 

Sex o? subjects was not investi gable since the overwhelming 
majority of studies (all but two) had mixed sex groups. Grade or 
age level of subjects U significantly related to effect (F=5.09, 
df ■ 6, 103; p < .05) but the significance is due to two cognitive 
outcomes observed with preschool children. In the absence of these 
effects overall, grade differences and differences within cognitive 
and attitude categories are all nonsignificant. 

Year of publication, sample size, and desiqn characteristics 
(report of reliability, random selection of subjects from population) 
were all nonsignificantly correlated with effect size. No multiple 
regressions were statistically or practically ( R-square = .05) 
significant. 

Discussion 

The results of this meta-analysis of pretest sensitization effects 
provide two conclusions which have implications for educational, psycho- 
logical and sociological research. The first is that there is a general 
pretest effect which cannot be safely ignored. Nonrandomized studies 
with pretests must be viewed with additional suspicion, since the data 
reported here are indicative of systematic bias due to pretest in such 
studies. 

The second conclusion about pretest effects is that they appear 
not to be uniform across psychological domains. The studies reported 
here are not sufficiently exhaustive to provide definitive statements 
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about conditions for variation of pretest sensitization, but promis- 
ing areas of investigation are change in cognitive effects over 
grade and age levels* transfer of pretest effect to different post- 
tests, and change in effect over delay time between pretest and post- 
test from hours to months. 
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Table 1 



Summary Statistics for Pretest 
Effect Size Distribution 



All 

Studies 



N Mean S.D. 



140 



.26 1.02 



Percentage 
Positive 
Outcomes 
67 



Max P 90 P 50 P 10 Min 



4.46 1.15 .09 -.50 -2.35 



Non- 
randomized 
Studies 



30 -.04 .63 



57 



1.15 .44 .04 -.61 -2.35 



Randomized 
Studies 110 



.34 1.09 



69 



4.64 1.6 .09 -.50 -2.30 
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Table 2 

Summary Statistics of Randomized Study 
Effects for Selected Independent Variables 



N Mean S.D. 



Treated 

Groups 70 .31* 1.06 



Percentage Max Pq Q Pr 0 

Positive 

Outcome 



10 



73% 



Min 



4.16 1.60 .06 -.50 -2.30 



Untreated 
Groups 



40 .39* 1.14 



65% 



4.46 1.41 .15 -.51 -1.63 



Duration 

Hours 
Days 
Weeks 
Months 



N 



30 
38 
15 
26 



Mean 



.25 
.44* 
.81* 
.11 



Omnibus Mean Square 
F-Statistic Error 



1.57 



1.12 



Outcome 
Category 

Cognitive 

Attitude 

Other 



27 
58 
25 



.51* 
.33* 
.17 



62 



1.19 



Pretest-Posttest 
Same 

Different 



83 
27 



.29 
.47* 



56 



1.19 



* P < .05, Ho: Mean = 0 
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N 



Grade/Age of 
Subjects 



Correlation with Effect 

Year of Publication 

Sample Size 

Reliability of Outcome 
reported? (l=yes 0=no 



Mean 



Adult 


8 


.20 


College 


74 


.34 


High School 


4 


.18 


Grade 11 


12 


.04 


Grade 9 


8 


.12 


Grade 4 


2 


.10 


Pre- School 


2 


4.06 
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Omnibus Mean Square 

F-Statistic Error 



5.09 



N 

110 

no 
no 



.969 



R 

.008 
-.128 

-.078 



Random selection of 
Subjects from Population? 
(l=yes, 0=no) 



110 



-.095 



-erIc 
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Table 3 

Summary statistics of Randomized 
Study Effects for Cognitive and 
Attitude Outcomes 



Cognitive Outcomes Attitude Outcomes 

N Mean N Mean 



Duration 



Hours 


16 


.28 


9 


.03 


Days 


2 


.17 


30 


.58* 


Weeks 


2 


4.06* 


13 


.31 


Months 


7 


.10 


6 


-.13 


Pretest-Posttest 










Same 


20 


.63* 


41 


.18 


Different 


7 


.15 


17 


.68* 


Treatment 










Treated Group 


22 


.39 


32 


.37 


Untreated Group 


5 


1.01* 


26 


.27 


Grade/Age of Subjects 










Adult 






2 


-.35 


College 


10 


.39 


52 


.38 


High School 


4 


.18 






Grade 11 






4 


-.02 


Grade 9 


8 


.12 


— •» 




Grade * 


2 


.10 


«• 




Pre- school 


2 


4.04* 


*» 





* P < .05; Ho: Mean = 0 



